What personal traits are inferred automatically?

Facial expressions as perceived by a model by (Peterson et al. 2022)

Don’t worry about the future…

The narrative of autonomous and evil-minded robots is an illusion.

…worry about the present

• AI is persuasive in everyday life
  • assessing risks and performances (credit, job, crime, terrorism etc.) (Hofmann et al. 2024)
• AI is extremely capable
  • increasingly difficult where it fails
• AI has data-driven bias
  • systems are often evaluated poorly

From bag of words to embeddings

Putting words into computers (Smith 2020; Church and Liberman 2021; Manning 2022)

• from coarse+static to fine+contextual meaning
• how to measure similarity of words and documents?
• from counting to learning representations

Bag of words

• word as arbitrary, discrete numbers
  • King = 1, Queen = 2, Man = 3, Woman = 4
• intrinsic meaning
• how are these words similar?

Vector-representations of words as discrete symbols (Colyer 2016)

Representing a corpus

“I eat ___ tonight”.

“The pizza was ___.”

You shall know a word by the company it keeps!

Firth (1957)

Formalize the linguistic intuition

1. mask words
2. let the computer predict them using its context

Word embeddings

word2vec (Mikolov et al. 2013)

• words as continuous vectors
  • accounting for similarity between words
• semantic similarity
  • King – Man + Woman = Queen
  • France / Paris = Switzerland / Bern

Contextualized word embeddings

BERT (Devlin et al. 2019)

• recontextualize static word embedding
  • different embeddings in different contexts
  • accounting for ambiguity (e.g., bank)
• acquire linguistic knowledge from loads of data
  • mask random phrases diverse sentences

From embeddings to generation

Instead of masking, train the model to predict the next word

Autocompletion on iPhone

It is a generic problem solver

Any task can be modeled as Text-to-Text

(Raffel et al. 2020)

Large Language Models (LLM) 💥

ChatGPT most successful, yet not unique

• scale up approach of predicting the next word
  • bigger models and more data
• optimize for dialogue instead of prose text
  • instruction-tuning (summarize, translate, reason)
  • Reinforcement Learning from Human Feedback (RLHF)

Associations in data

«___ becomes a doctor.»

Learning patterns from data

Gender bias of the commonly used language model BERT (Devlin et al. 2019)

Cultural associations in training data

Gender bias of the commonly used language model BERT (Devlin et al. 2019)

Word embeddings are biased …

… because our data is we are biased. (Bender et al. 2021)

In-class: Exercises I

1. Open the following website in your browser: https://pair.withgoogle.com/explorables/fill-in-the-blank/
2. Read the the article and play around with the interactive demo.
3. What works surprisingly well? What looks flawed by societal bias? Where do you see limits of large language models?

How does deep learning model look like?

Simplified illustration of a neural network. Arrows are weights and bullets are (intermediate) states.

How does deep learning work?

Current state of deep learning

Extremely powerful but … (Bengio, Lecun, and Hinton 2021)

• great at learning patterns, yet reasoning in its infancy
• requires tons of data due to inefficient learning
• generalizes poorly

Limitations of data-driven deep learning

Biased data with practical implications

Gender bias in Google Translate

Raw data is an oxymoron.

Gitelman (2013)

Fair is a fad

• companies also engage in fair AI to avoid regulation
• Fair and good – but to whom? (Kalluri 2020 )
• lacking democratic legitimacy

Don’t ask if artificial intelligence is good or fair, ask how it shifts power.

Kalluri (2020)

Algorithmic managment of labour force

Text generation may be used to communicate difficult decisions strategically

Data represents real life.

Don’t be a fool. Be wise, think twice.

Additional resources

• deepen your understanding of modern LLMs with

  • this video

  • this animated explainer